--- title: Transformations for 3D medical images keywords: fastai sidebar: home_sidebar nb_path: "nbs/02_transforms.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %}

Resizing

Using the @patch decorator makes the transform function a callable for the Tensor class (and all subclasses, including TensorDicom3D and TensorMask3D).

{% raw %}

TensorDicom3D.resize_3d[source]

TensorDicom3D.resize_3d(t:TensorMask3D'>), size, scale_factor=None, mode='trilinear', align_corners=True, recompute_scale_factor=None)

A function to resize a 3D image using torch.nn.functional.interpolate

Args: t (Tensor): a 3D or 4D Tensor to be resized size (int): a tuple with the new x,y,z dimensions of the tensor after resize scale_factor, mode, align_corners, recompute_scale_factor: args from F.interpolate Returns: A new Tensor with Tensor.size = size

{% endraw %} {% raw %}

TensorMask3D.resize_3d[source]

TensorMask3D.resize_3d(t:TensorMask3D'>), size, scale_factor=None, mode='trilinear', align_corners=True, recompute_scale_factor=None)

A function to resize a 3D image using torch.nn.functional.interpolate

Args: t (Tensor): a 3D or 4D Tensor to be resized size (int): a tuple with the new x,y,z dimensions of the tensor after resize scale_factor, mode, align_corners, recompute_scale_factor: args from F.interpolate Returns: A new Tensor with Tensor.size = size

{% endraw %} {% raw %}

index_based_resize[source]

index_based_resize(t:TensorMask3D, size:int)

resizes a Tensor without creating new values

{% endraw %} {% raw %}

class Resize3D[source]

Resize3D(size, scale_factor=None, mode='trilinear', align_corners=True, recompute_scale_factor=None, **kwargs) :: RandTransform

A transform that before_call its state at each __call__

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
Resizer = Resize3D((10,50,50))
{% endraw %} {% raw %} 
original = TensorDicom3D.create('/media/ScaleOut/prostata/data/dcm/A0042197734/T2/DICOM')
mask = TensorMask3D.create('/media/ScaleOut/prostata/data/dcm/A0042197734/T2/Annotation/cropped_mask.nii.gz')
{% endraw %} {% raw %} 
original.show()
{% endraw %} {% raw %} 
mask.show()
{% endraw %} {% raw %} 
im = Resizer(original, split_idx = 0)
ma = Resizer(mask, split_idx = 0)
{% endraw %} {% raw %} 
im.show()
{% endraw %} {% raw %} 
ma.show()
{% endraw %}

Flipping

In medical images, the left and right side often cannot be differentiated from each other (e.g. scans of the head, hand, knee, ...). Therfore the image orientation is stored in the image header, enabeling the viewer system to accuratly display the images. For deep learning, only the pixel array is extracted, so the header information is lost. When displaying only the pixel array, the images might already be displayed flipped or in inverted slice order. So, implementing vertical/horizontal flipping as well as flipping alongside the z axis can be used for data augmentation.

{% raw %}

TensorDicom3D.flip_ll_3d[source]

TensorDicom3D.flip_ll_3d(t:TensorMask3D'>))

flips an image laterolateral

{% endraw %} {% raw %}

TensorMask3D.flip_ll_3d[source]

TensorMask3D.flip_ll_3d(t:TensorMask3D'>))

flips an image laterolateral

{% endraw %} {% raw %}

TensorDicom3D.flip_ap_3d[source]

TensorDicom3D.flip_ap_3d(t:TensorMask3D'>))

flips an image anterior posterior

{% endraw %} {% raw %}

TensorMask3D.flip_ap_3d[source]

TensorMask3D.flip_ap_3d(t:TensorMask3D'>))

flips an image anterior posterior

{% endraw %} {% raw %}

TensorDicom3D.flip_cc_3d[source]

TensorDicom3D.flip_cc_3d(t:TensorMask3D'>))

flips an image cranio caudal

{% endraw %} {% raw %}

TensorMask3D.flip_cc_3d[source]

TensorMask3D.flip_cc_3d(t:TensorMask3D'>))

flips an image cranio caudal

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
show_images_3d(torch.stack((im, flip_ll_3d(im), flip_ap_3d(im), flip_cc_3d(im))))
{% endraw %} {% raw %}

class RandomFlip3D[source]

RandomFlip3D(p=0.75) :: RandTransform

Randomly flip alongside any axis with probability p

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
flipper = RandomFlip3D()
flipper(im)
show_images_3d(torch.stack((im, flipper(im, split_idx = 0), flipper(im, split_idx = 0), flipper(im, split_idx = 0))))
{% endraw %}

Rotating

Medical images should show no rotation, however with removal of the image file header, the pixel array might appear rotated when displayed and thus be introduced to the model rotated. Fruthermore, in some images the patients might be rotated to some degree. Thus rotation of 90 and 180° as well as substeps should be implemented.

{% raw %}

TensorDicom3D.rotate_90_3d[source]

TensorDicom3D.rotate_90_3d(t:TensorMask3D'>))

{% endraw %} {% raw %}

TensorMask3D.rotate_90_3d[source]

TensorMask3D.rotate_90_3d(t:TensorMask3D'>))

{% endraw %} {% raw %}

TensorDicom3D.rotate_270_3d[source]

TensorDicom3D.rotate_270_3d(t:TensorMask3D'>))

{% endraw %} {% raw %}

TensorMask3D.rotate_270_3d[source]

TensorMask3D.rotate_270_3d(t:TensorMask3D'>))

{% endraw %} {% raw %}

TensorDicom3D.rotate_180_3d[source]

TensorDicom3D.rotate_180_3d(t:TensorMask3D'>))

{% endraw %} {% raw %}

TensorMask3D.rotate_180_3d[source]

TensorMask3D.rotate_180_3d(t:TensorMask3D'>))

{% endraw %} {% raw %}

class RandomRotate3D[source]

RandomRotate3D(p=0.5) :: RandTransform

Randomly flip rotates the axial slices of the 3D image 90/180 or 270 degrees with probability p

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
rotator = RandomRotate3D()
show_images_3d(torch.stack((im, rotate_90_3d(im), rotate_180_3d(im), rotate_270_3d(im), 
                                   rotator(im, split_idx = 0),  rotator(im, split_idx = 0),  rotator(im, split_idx = 0))))
{% endraw %} {% raw %}

TensorDicom3D.rotate_3d_by[source]

TensorDicom3D.rotate_3d_by(t:TensorMask3D'>), angle:(<class 'int'>, <class 'float'>), axes:list)

rotates 2D slices of a 3D tensor. Args: t: a TensorDicom3D object or torch.tensor angle: the angle to rotate the image axes: axes to which the rotation should be applied.

Example: If the tensor t has the shape of (10, 512, 512), which is equal to 10 slices of 512x512 px.

rotate_3d_by(t, angle = -15.23, axes = [1,2]) will rotate each slice for -15.23 degrees.

BUG: Does not work properly on CUDA

{% endraw %} {% raw %}

TensorMask3D.rotate_3d_by[source]

TensorMask3D.rotate_3d_by(t:TensorMask3D'>), angle:(<class 'int'>, <class 'float'>), axes:list)

rotates 2D slices of a 3D tensor. Args: t: a TensorDicom3D object or torch.tensor angle: the angle to rotate the image axes: axes to which the rotation should be applied.

Example: If the tensor t has the shape of (10, 512, 512), which is equal to 10 slices of 512x512 px.

rotate_3d_by(t, angle = -15.23, axes = [1,2]) will rotate each slice for -15.23 degrees.

BUG: Does not work properly on CUDA

{% endraw %} {% raw %}

class RandomRotate3DBy[source]

RandomRotate3DBy(p=0.5) :: RandTransform

Randomly flip rotates the axial slices of the 3D image 90/180 or 270 degrees with probability p

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
rotator_by = RandomRotate3DBy()
{% endraw %} {% raw %} 
show_images_3d(torch.stack((im, rotate_3d_by(im, angle = 15, axes = [1,2]), rotator_by(im, split_idx = 0))))
{% endraw %} {% raw %} 
im2 = resize_3d(readdcm_3d('/media/ScaleOut/prostata/data/dcm/A0042197734/T2/DICOM', return_normalized = True), (25, 10, 50))
show_images_3d(torch.stack((im2, rotate_3d_by(im2, angle =10, axes = [0,2]))), axis =1, nrow = 5)
{% endraw %}

Rotating by 90 (or 180 and 270) degrees should not be done via rotate_3d_by but by rotate_90_3d, is approximatly 28 times faster.

%%timeit rotate_3d_by(im, angle = 90, axes = [1,2])%%timeit rotate_90_3d(im2)

Dihedral transformation

As the 3D array can be flipped by three sides, but should only be rotated along the z axis, this is not a complete dihedral group. Still multiple combinations of flipping and rotating should be implemented:

  1. original (= flipp ll, roate 180 = same as original image)
  2. rotate 90
  3. rotate 180
  4. rotate 270
  5. flip ll (=flip ap, rotate 180)
  6. flip ap
  7. flip cc
  8. flip cc, rotate 90
  9. flip cc, rotate 180
  10. flip cc, rotate 270
  11. flip ll, rotate 90
  12. flipp ll, rotate 270
  13. flip ap, rotate 90
  14. flip ap rotate 270
  15. flip cc, flip ll, rotate 90
  16. flip cc, flipp ll, rotate 270
  17. flip cc, flip ap, rotate 90
  18. flip cc, flip ap rotate 270

I am not sure if this is complete...

{% raw %}

TensorDicom3D.dihedral3d[source]

TensorDicom3D.dihedral3d(x:TensorMask3D'>), k)

apply dihedral transforamtions to the 3D Dicom Tensor

{% endraw %} {% raw %}

TensorMask3D.dihedral3d[source]

TensorMask3D.dihedral3d(x:TensorMask3D'>), k)

apply dihedral transforamtions to the 3D Dicom Tensor

{% endraw %} {% raw %}

class RandomDihedral3D[source]

RandomDihedral3D(p=1.0, nm=None, before_call=None, **kwargs) :: RandTransform

randomly flip and rotate the 3D Dicom volume with a probability of p

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
dihedral = RandomDihedral3D()
show_images_3d(torch.stack((im, dihedral(im, split_idx = 0), dihedral(im, split_idx = 0), 
                                    dihedral(im, split_idx = 0),dihedral(im, split_idx = 0), 
                                    dihedral(im, split_idx = 0))))
{% endraw %}

Random Crop

A reasonable approach for 3D medical images would be a presizing to uniform but to large volume and subsequent random cropping to the target dimension. As most areas of interest are located centrally in the image/volume some cropping can always be applied.
Also random cropping should be applied after any rotation, that is not in 90/180/270 degrees, so that empty margins are cropped.

{% raw %}

TensorDicom3D.crop_3d[source]

TensorDicom3D.crop_3d(t:TensorMask3D'>), crop_by:(<class 'int'>, <class 'float'>), perc_crop=False)

Similar to function crop_3d_tensor, but no checking for margin formats is done, as they were correctly passed to this function by RandomCrop3D.encodes

{% endraw %} {% raw %}

TensorMask3D.crop_3d[source]

TensorMask3D.crop_3d(t:TensorMask3D'>), crop_by:(<class 'int'>, <class 'float'>), perc_crop=False)

Similar to function crop_3d_tensor, but no checking for margin formats is done, as they were correctly passed to this function by RandomCrop3D.encodes

{% endraw %} {% raw %}

class RandomCrop3D[source]

RandomCrop3D(crop_by, rand_crop_xyz, perc_crop=False, p=1, **kwargs) :: RandTransform

Randomly crop the 3D volume with a probability of p The x axis is the "slice" axis, where no cropping should be done by default

Args crop_by: number of pixels or pecantage of pixel to be removed at each side. E.g. if (0, 5, 5), 0 pixel in the x axis, but 10 pixels in eacht y and z axis will be cropped (5 each side) rand_crop_xyz: range in which the cropping window is allowed to vary. perc_crop: if true, no absolute but relative number of pixels are cropped

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
Cropper = RandomCrop3D((0,10,10), (0,5,5), False)

show_images_3d(torch.stack((Cropper(im, split_idx = 0), Cropper(im, split_idx = 0), 
                                    Cropper(im, split_idx = 0), Cropper(im, split_idx = 0))))
{% endraw %}

Other cropping methods, with padding, squishing or large maginifcations might not be appropriate for medical images, since often only small areas in the image are of importance which could be removed by cropping (e.g. tumor). So cropping should only be applied to the image margins.

As cropping a resizing are good preprocessing operations, they can be merged into one class, for easier access.

{% raw %}

class ResizeCrop3D[source]

ResizeCrop3D(crop_by, resize_to, perc_crop=False, p=1, **kwargs) :: RandTransform

A transform that before_call its state at each __call__

{% endraw %} {% raw %}
{% endraw %}

Warping

Light wrapping can mimic artifacts in MRI images or breathing artifacts in CT images

{% raw %}

TensorDicom3D.warp_3d[source]

TensorDicom3D.warp_3d(t:TensorMask3D'>), magnitude_x, magnitude_y)

A function to warp a 3D image using torch.nn.functional.grid_sample

Taken form the offical documention: Given an input and a flow-field grid, computes the output using input values and pixel locations from grid. In the spatial (4-D) case, for input with shape (N,C,Hin,Win) and with grid in shape (N, Hout, Wout, 2), the output will have shape (N, C, Hout,Wout)

In the case of 5D inputs, grid[n, d, h, w] specifies the x, y, z pixel locations for interpolating output[n, :, d, h, w].
mode argument specifies nearest or bilinear interpolation method to sample the input pixels.

Workflow of this function:

  1. create a fake RGB 3D image through generating fake color channels.
  2. add a 5th batch dimension
  3. create a flow-field for rescaling: a. creat2 two 1D tensor giving a linear progression from -1 to 0 and 0 to 1 with differnt number of steps, then merge them to one tensor b. creat a mesh-grid (the flow field) from x,y,z tensors from (a)
  4. resample the input tensor according to the flow field
  5. remove fake color channels and batch dim, returning only the 3D tensor

Args: t (Tensor): a Rank 3 Tensor to be resized

{% endraw %} {% raw %}

TensorMask3D.warp_3d[source]

TensorMask3D.warp_3d(t:TensorMask3D'>), magnitude_x, magnitude_y)

A function to warp a 3D image using torch.nn.functional.grid_sample

Taken form the offical documention: Given an input and a flow-field grid, computes the output using input values and pixel locations from grid. In the spatial (4-D) case, for input with shape (N,C,Hin,Win) and with grid in shape (N, Hout, Wout, 2), the output will have shape (N, C, Hout,Wout)

In the case of 5D inputs, grid[n, d, h, w] specifies the x, y, z pixel locations for interpolating output[n, :, d, h, w].
mode argument specifies nearest or bilinear interpolation method to sample the input pixels.

Workflow of this function:

  1. create a fake RGB 3D image through generating fake color channels.
  2. add a 5th batch dimension
  3. create a flow-field for rescaling: a. creat2 two 1D tensor giving a linear progression from -1 to 0 and 0 to 1 with differnt number of steps, then merge them to one tensor b. creat a mesh-grid (the flow field) from x,y,z tensors from (a)
  4. resample the input tensor according to the flow field
  5. remove fake color channels and batch dim, returning only the 3D tensor

Args: t (Tensor): a Rank 3 Tensor to be resized

{% endraw %} {% raw %}

Tensor.warp_4d[source]

Tensor.warp_4d(t:Tensor, magnitude_x, magnitude_y)

{% endraw %} {% raw %}

Tensor.warp_4d[source]

Tensor.warp_4d(t:Tensor, magnitude_x, magnitude_y)

{% endraw %} {% raw %}

class RandomWarp3D[source]

RandomWarp3D(p=0.5, max_magnitude=0.25) :: RandTransform

A transform that before_call its state at each __call__

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
warper = RandomWarp3D(p=1)
show_images_3d(torch.stack((warper(im, split_idx = 0), warper(im, split_idx = 0), warper(im, split_idx = 0), warper(im, split_idx = 0), warper(im, split_idx = 0))))
{% endraw %}

Random Gaussian noise

Older scanners with lower field strength (MRI), fewer slices (CT), older algorithms can be more noise. So adding some random noise to the data, could improve model performance.

{% raw %}

TensorDicom3D.add_gaussian_noise[source]

TensorDicom3D.add_gaussian_noise(t:TensorDicom3D, std)

{% endraw %} {% raw %}

class RandomNoise3D[source]

RandomNoise3D(p=0.5) :: RandTransform

A transform that before_call its state at each __call__

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
noise_adder = RandomNoise3D(p=1)
show_images_3d(torch.stack((noise_adder(im, split_idx = 0), noise_adder(im, split_idx = 0), 
                                     noise_adder(im, split_idx = 0), noise_adder(im, split_idx = 0), noise_adder(im, split_idx = 0))))
{% endraw %}

Lightning transforms

Simple brightness and contrast controlls can be dona via a linear function:

x = alpha * x_i + beta

here x_i is the respective pixel, alpha allows simple contrast control, beta allows simple brightness control.

{% raw %}

TensorDicom3D.rescale[source]

TensorDicom3D.rescale(t:TensorDicom3D, new_min=0, new_max=1)

{% endraw %} {% raw %}

TensorDicom3D.adjust_brightness[source]

TensorDicom3D.adjust_brightness(x:TensorDicom3D, beta)

{% endraw %} {% raw %}

class RandomBrightness3D[source]

RandomBrightness3D(p=0.5) :: RandTransform

A transform that before_call its state at each __call__

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
lighting = RandomBrightness3D()
show_images_3d(torch.stack((im, lighting(im, split_idx = 0), lighting(im, split_idx = 0), 
                                     lighting(im, split_idx = 0), lighting(im, split_idx = 0), 
                                     lighting(im, split_idx = 0), lighting(im, split_idx = 0))))
{% endraw %}

Contrast

{% raw %}

TensorDicom3D.adjust_contrast[source]

TensorDicom3D.adjust_contrast(x:TensorDicom3D, alpha)

{% endraw %} {% raw %}

class RandomContrast3D[source]

RandomContrast3D(p=0.6) :: RandTransform

A transform that before_call its state at each __call__

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
contrast = RandomContrast3D()
show_images_3d(torch.stack((im, contrast(im, split_idx = 0), contrast(im, split_idx = 0), 
                                     contrast(im, split_idx = 0), contrast(im, split_idx = 0), 
                                     contrast(im, split_idx = 0), contrast(im, split_idx = 0))))
{% endraw %}

Random ereasing

Random ereasing, as shown in the fastaivision.augment docs might not be optimal, because it could erease the important image findings, such as a nodule. Just ereasing random pixel could be an option, but I belive this would not be different from using dropout while training a model?

Putting it all together

A good workflow would be to apply random crop to all images after one transformation. For this, the images should be presized to a size, just some pixels larger then desired, then transformed and then cropped to the final size. Using this approach empty space, which e.g. appears after RandomRotate3DBy will be cropped and not influence the accuracy of the model. One only has to be careful, that the region of interest, e.g. the prostate, will be in every cropped image.

{% raw %} 
im = readdcm_3d('/media/ScaleOut/prostata/data/dcm/A0042197734/T2/DICOM', return_normalized = True)
im = resize_3d(im, (30, 250, 250)) # presizing the images

Cropper = RandomCrop3D((5,50,50), (1,5,5))

tfms = [RandomBrightness3D(), RandomContrast3D(), RandomWarp3D(), RandomDihedral3D(), RandomNoise3D(), RandomRotate3DBy()]
tfms = [Pipeline([RandomBrightness3D, Cropper], split_idx = 0), 
        Pipeline([RandomContrast3D, Cropper], split_idx = 0), 
        Pipeline([RandomWarp3D, Cropper], split_idx = 0), 
        Pipeline([RandomDihedral3D, Cropper], split_idx = 0), 
        Pipeline([RandomNoise3D, Cropper], split_idx = 0), 
        Pipeline([RandomRotate3DBy, Cropper], split_idx = 0)]
{% endraw %} {% raw %} 
comp = setup_aug_tfms(tfms)
{% endraw %} {% raw %} 
ims = [t(im) for t in tfms]
show_images_3d(torch.stack(ims))
{% endraw %}

Creating a pseudo color channel

Pytorch expects the images in the following format:

B C D H W

Here:

  • B = Batch dimension
  • C = Number of Channels (e.g. color)
  • D = Depth of the image (= number of slices)
  • H = Height of the image
  • W = Width of the image
{% raw %}

TensorDicom3D.make_pseudo_color[source]

TensorDicom3D.make_pseudo_color(t:TensorMask3D'>))

The 3D CNN still expects color images, so a pseudo color image needs to be created as long as I don't adapt the 3D CNN

{% endraw %} {% raw %}

TensorMask3D.make_pseudo_color[source]

TensorMask3D.make_pseudo_color(t:TensorMask3D'>))

The 3D CNN still expects color images, so a pseudo color image needs to be created as long as I don't adapt the 3D CNN

{% endraw %} {% raw %}

class PseudoColor[source]

PseudoColor(p=1) :: RandTransform

A transform that before_call its state at each __call__

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
MakeColor = PseudoColor()
im.shape, MakeColor(im, split_idx = 0).shape
{% endraw %} {% raw %}
{% endraw %} {% raw %}

aug_transforms_3d[source]

aug_transforms_3d(p_all=0.1, do_warp=True, p_warp=None, do_dihedral=True, p_dihedral=None, do_brightness=True, p_brightness=None, do_contrast=True, p_contrast=None, do_noise=True, p_noise=None, do_rotate_by=True, p_rotate_by=None, do_flip=True, p_flip=None, do_rotate=True, p_rotate=None, do_crop=True, p_crop=1)

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
aug_transforms_3d()
{% endraw %}

Mask Transformations

Transformations for the mask in segmentation tasks. If it is a multilabel segmentation task, the mask needs to be converted into a one hot encoded tensor.

{% raw %} 
# Is now implemented as a callback

def _make_binary(t, set_to_one):
    "Sets all but one values to zero. The remaining value is set to one."
    return (t == set_to_one).float().to(t.device)


@patch
def to_one_hot(m:(Tensor,TensorMask3D), num_features:int):
    """
    Takes a Tensor and will return a one hot encoded version, 
    where every layer of the 2nd channel corresponds to a single 
    one hot encoded value.
    
    Args: 
        m: a Tensor or TensorMask3D in the Format: B*C*D*H*W where C should be 1
        num_features: number of features to be one_hot_encoded
        
    Returns: 
        A one hot encoded tensor with the number of color channels corresponding to num_features
    """
    m = m.squeeze(1).long() # remove the solitary color channel (if there is one) and set type to int64
    one_hot = [_make_binary(m, set_to_one=i) for i in range(0, num_features + 1)]
    
    return torch.stack(one_hot, 1).to(m.device)

class MaskOneHot(RandTransform):
    split_idx, p = 1, 1
    
    def __init__(self, p=1): 
        super().__init__(p=p)

    def __call__(self, b, split_idx=1, **kwargs):
        "change in __call__ to enforce, that the Transform is always applied on every dataset. "
        return super().__call__(b, split_idx=split_idx, **kwargs) 
    
    def encodes(self, x:(TensorMask3D)): 
        return x.to_one_hot()
{% endraw %} {% raw %}

TensorMask3D.clamp_to_range[source]

TensorMask3D.clamp_to_range(x:TensorMask3D, lwr, upr)

{% endraw %} {% raw %}

class ClampMask3D[source]

ClampMask3D(lwr=0, upr=1, p=1) :: RandTransform

Clamps/Clips mask value to a range

{% endraw %} {% raw %}
{% endraw %}